Degenerate Principal Series Representations and Their Holomorphic Extensions

Let X = H/L be an irreducible real bounded symmetric domain realized as a real form in an Hermitian symmetric domain D = G/K. The intersection S of the Shilov boundary of D with X defines a distinguished subset of the topological boundary of X and is invariant under H and can also be realized as S = H/P for certain parabolic subgroup P of H . We study the spherical representations Ind P (λ) of H induced from P . We find formulas for the spherical functions in terms of the Macdonald 2F1 hypergeometric function. This generalizes the earlier result of FarautKoranyi for Hermitian symmetric spaces D. We consider a class of H-invariant integral intertwining operators from the representations Ind P (λ) on L(S) to the holomorphic representations of G on D restricted to H . We construct a new class of complementary series for the groups H = SO(n,m), SU(n,m) (with n−m > 2) and Sp(n,m) (with n−m > 1). We realize them as a discrete component in the branching rule of the analytic continuation of the holomorphic discrete series of G = SU(n,m), SU(n,m)× SU(n,m) and SU(2n, 2m) respectively.